30/11/23 - Finalising site-capabilities API changes

29/11/23 - Finalising site-capabilities API changes

Also, annual review.

Demo on ingestion service.

Amending site-capabilites with concept of processing, storage and associated services.

amended postgres_meta.py and added schemas
fixed issue with lax literal_eval in earlier versions of python (https://github.com/rucio/rucio/issues/6374)
added better error handling for psycopg2 connections to avoid cursor getting lost (https://github.com/rucio/rucio/issues/6372)

problem with get_site (version==latest?) fixed
services have an associated storage id field (optional)
added route for dumping latest to document for db
addd site ls (requires site-capabilities-client changes)
added client endpoint that returns www urls for site add/edit.
fixed double printing SKAOSRC storage ? (list_storages)
added add/amend site entrypoints on ska-src-clients that prints url (maybe opens browser?) with ?token= query populated
added site ls to ska-src-clients
changed get upload logic token to use associated_storage_id
changed soda logic to use associated_storage_id
added schema render endpoint for plantuml;

DM now gives token access to /ingest/staging/{namespace} (site-cap change).
Implemented oidc token for services in Dockerfile of ingest service (gets a service token automatically)

Still need to change the docker base to the ska rucio client image.

Finished upload flow in client.

Restarting cloud machines, checking everything's ok. Finalising the upload-to-staging area logic.